(1) Field of the Invention
The present invention relates to image processing as well as a display technique for still images. In particular, the present invention to a technique of controlling an area in view of an object to be extracted when displaying the image as well as a technique of generating and replaying a short film with the use of at least one still image.
(2) Description of the Related Art
In 2001, the number of shipments of digital steel cameras (hereafter to be referred to as “digital camera”) has surpassed that of silver salt cameras in the Japanese market for cameras, and it is predicted that the difference in number will further increase in the future. The recent market for cell phones has released a number of cell phones, each being equipped with a digital camera and these models are gaining a huge popularity. Thus, the number of users using a digital camera grows steadily.
On the other hand, a portable terminal (e.g., a cell phone) equipped with a function to display an image is also widely used to the extent that one person carries one cell phone, and the demand for displaying the images shot by a digital camera on an image display apparatus of the portable terminal (hereafter to be referred to as “image display apparatus”) grows day by day.
An image formation apparatus like a digital camera, in general, shoots an object with the size of 1280×960 pixels and stores, in a medium such as an SD memory card (™), the image with the size of 1280×960 pixels (to be referred to as “original image”) and a thumbnail image with the size of 160×120 pixels in EXIF format. The display of the image display apparatus, however, displays an image with the size of 240×320 pixels (QVGA) at maximum.
Thus, in the case where there is a difference between the resolution of the image to be displayed by the display of the image display apparatus and the resolution of the image to be displayed by the image display apparatus (referred to as “display image”), it is necessary to scale the display image in accordance with either of the resolutions.
Therefore, in the case of cell phone, the method to judge whether the image to be displayed can be displayed within a display area, and display the thumbnail image that is separately stored in the case where the original image does not fit in the display area is suggested as a technique related to the conventional image formation apparatus (see reference to Japanese Laid-Open Patent Publication 2002-23914).
Along with the wide use of digital cameras and cell phones with a camera, an increase in the number of users keeping huge amount of digital still images (hereafter to be referred to as “stillimages”) is remarkable. These still images, in general, are stored in a disk drive in the personal computer. A short film generation apparatus for generating a short film like a movie based on still images and replaying it is suggested as an apparatus to replay such a huge amount of still images. For example, a technique to replay a still image while applying a visual effect to it based on the contents of dynamic image processing and a procedure in time processing (i.e., scenario) is disclosed in Japanese Laid-Open Application H 11-15993.
FIG. 1 is a block diagram showing the configuration of the conventional short film generation apparatus.
The conventional short film generation apparatus 900 consists of an operation receiving unit 901, an image receiving (obtainment) unit 902, an image accumulation unit 903, a scenario generation unit 904, an image replay unit 905 and a display unit 906.
The operation receiving unit 901 receives the inputs such as an instruction for reading still image data or a selection of still images operated by the user.
The image receiving unit 902 reads in the still image data in JPEG compressive format taken by a digital camera 910.
The image accumulation unit 903 accumulates the read-in still image data.
The scenario generation unit 904 decides an order of replaying the still images and respective visual effects, based on the still images selected by the user, generates a scenario that is replay information and stores it in the memory. Here, a scenario is not moving image data itself as represented by MPEG, but rather the data in which (i) a visual effect (effect), (ii) a parameter to define the operation of the effect, (iii) a pointer pointing at the still images to which the visual effect is applied, and (iv) a replay time are described in chronological order using a script language as represented by XML.
The image replay unit 905 repeats the following operation: reading out still image data from the image accumulation unit 903 according to the generated scenario; and replaying it with a speed of 15 frames/second while applying the predetermined visual effect.
The display unit 906 displays a user interface screen or an image to be replayed under the control of the scenario generation unit 904 and the image replay unit 905.
The short film generation apparatus 900 constructed as above operates as follows.
Firstly, the image shot by the user with the digital camera 910 is read in by the short film generation apparatus 900 via the image receiving unit 902 and then transferred and stored in the image accumulation unit 903.
Here, when the operation receiving unit 901 receives the user's operation to press a short film generation button, the scenario generation unit 904 generates a scenario.
FIG. 2 is a flowchart showing a procedure of the processing performed by the conventional scenario generation unit 904.
Firstly, the scenario generation unit 904 generates a screen (an image selection screen) to urge the user to select plural images stored in the image accumulation unit 903 and displays the screen on the display unit 906 (S900).
Here, when the operation receiving unit 901 receives the user's operation to select the still image to be replayed from the images to be displayed by the display unit 906 (Yes in S902), the scenario generation unit 904 checks the shooting time information is of the selected image and generates a scenario frame which links lists the replay information for each of the images in chronological order (S904).
Here, FIG. 3 shows the scenario frame generated in the case where the operation receiving unit 901 receives a selection of five still images (images 31, 32, 33, 34 and 35) from the user. It should be noted that the shooting time of the five still images are arranged in chronological order of images 31, 32, 33, 34 and 35.
FIG. 3 shows the conventional scenario frame.
The conventional scenario frame is like a linked list consisting of a scenario header 920, a scenario first list 921, a scenario second list 922, a scenario third list 923, a scenario fourth list 924 and a scenario fifth list 925.
In this case, a scenario name and a number of images are set in the scenario header 920 while an image file name and a storing place, out of the replay information for each image, are set in each of the lists 921˜925 in the scenario frame.
A total replay time in the scenario header 920 and visual effect information and replay time information in each of the lists 921˜925 in the scenario frame are left vacant.
The scenario generation unit 904 decides the visual effect to be assigned for each image in order of replaying images after having generated such a scenario.
As shown in FIG. 2, the scenario generation unit 904 firstly refers to the scenario first list 921 in the scenario frame (S906) and decides the visual effect to be assigned for the image 31.
Here, the scenario generation unit 904 generates a random number using a random function, divides it by 4 (S908) and decides the visual effect to be assigned for the image 31 by obtaining the remainder.
Namely, the scenario generation unit 904 adds, to the visual effect information in the scenario first list 921, “fade-in” for gradually clarifying the image (S912), in the case where the remainder is “1” (Yes in S910), and adds, to the visual effect information in the scenario first list 921, “fade-out” for gradually removing the image, in the case where the remainder is “2” (S916).
The scenario generation unit 904 adds, to the visual effect information in the scenario first list 921, “zoom-in” for enlarging the image in the center direction (S920), in the case where the remainder is “0” (Yes in S918), and adds, to the visual effect information in the scenario first list 921, “Pan” for displaying larger the part of the image and shifting the displayed part (S922), in the case where the remainder is “3” (No in S918).
The scenario generation unit 904 further sets the replay time of the image and adds the replay time information to the scenario first list 921, when adding the visual effect to be assigned to the visual effect information in the scenario first list 921. Here, the replay time of the image to which the zoom-in effect is applied is set as three seconds (45 frames) while two seconds is set for other images (30 frames).
After that, the scenario generation unit 904 judges whether or not the list to which the visual effect information and the replay time information are added is the last list (S924). The scenario generation unit 904 terminates the operation of scenario generation in the case where the list is the last list (Yes in S924) but refers to the next list in the scenario frame (S926) and repeats the operation to decide the visual effect, in the case where the list is not the last list (No in S924).
In this way, the scenario shown in FIG. 4 is completed after the repetition of the operation to assign the visual effect performed by the scenario generation unit 904 until it comes to the last image in replay order.
FIG. 4 shows the first example of the conventional completed scenario.
In the completed scenario, the information indicating 12 seconds for the total replay time is described in the scenario header 930, the visual effect information for assigning the fade-in effect as well as 2 seconds for the replay time information are described in the scenario first list 931 in which the replay information of the image 31 is described. In the same way, the visual effect information and the replay time information to be respectively assigned are described in the list where the replay information is described for each image.
That is to say, according to the completed scenario, the images 31, 32, 33, 34 and 35 are replayed in this order, and it is indicated that the fade-in effect is applied to the image 31, the zoom-out effect to the image 32, the fade-out effect to the image 33, the Pan effect to the image 34 and the zoom-in effect to the image 35. The respective replay time are indicated as 2 seconds, 3 seconds, 2 seconds, 2 seconds and 3 seconds (12 seconds in total).
Here, when the operation receiving unit 901 receives the input of the title of the short film from the user, in some cases, the scenario generation unit 904 inserts the replay information of the title image (an image with a black screen prepared to be used for the title) at the head of the list of the completed scenario.
In this case, the scenario generation unit 904 adds, to the head of the list of the scenario, the visual effect information indicating the insertion of the characters inputted by the user in the center of the image as well as the replay time information (2 seconds). FIG. 5 shows the scenario thus completed.
In the completed scenario, the title image, the images 31, 32, 33, 34 and 35 are replayed in this order and it is indicated that the character insertion effect is applied to the title image, the fade-in effect to the image 31, the zoom-out effect to the images 32 and the fade-out effect to the image 33, the Pan effect to the image 34 and the zoom-in effect to the image 35. The respective replay time are indicated as 2 seconds, 2 seconds, 3 seconds, 2 seconds, 2 seconds and 3 seconds (14 seconds in total).
Namely, the information indicating 14 seconds for the total replay time is described in the scenario header 940 while the visual effect information for assigning the character insertion effect for inserting the characters inputted by the user in the center of the image as well as replay time information indicating 2 seconds for the replay time as is described in the scenario first list 941 in which the replay information of the title image is described. In the same way, the visual effect information and the replay time information are respectively assigned in the respective list where the replay information is described for each image.
The image replay unit 905 receives address information of the scenario generated by the scenario generation unit 904 using the procedure described above and refers to the replay information of the image stored in the scenario first list.
The image replay unit 905 then reads out the image in compressive format from the image accumulation unit 903 based on the replay information that is already referred to and decompresses it. Then the image replay unit 905 applies the visual effect indicated in the visual effect information and outputs it to the display unit 906 while performing time management based on the replay time information (the number of frames). After the replay of the head image is completed, the image replay unit 905 executes the replay processing with reference to the replay information of the next image indicated in the scenario, and repeats the same processing until the replay of the last image (image 35) is terminated.
The following describes the visual effect of the image to be outputted for display on the display unit with reference to FIG. 6.
Here, the Pan effect for shifting in diagonal direction on the image is explained.
FIG. 6 shows an example of the screen display performed by the conventional short film generation apparatus 900.
According to the Pan effect in diagonal direction on the image, the short film generation apparatus 900 enlarges the first display area 951 for display after having displayed the display screen example 950, then, displays the second display area 953 by shifting the part to be displayed along with the display transition line 952, and then terminates the replay.
In this way, the conventional short film generation apparatus 900 generates a short film to be displayed as if an image were moving, and replays it while applying to the plural still images the visual effects such as zoom-in, fade-in, fade-out and Pan. It is innovative that such conventional short film generation apparatus 900 has opened up the possibility of new entertainment using digital still images by giving the still images the movements as seen in the moving image.
However, with the use of the conventional image display apparatus, in the case where the resolution (number of pixels) of the display image is higher (larger) than that of the display, it is necessary to store thumbnail images having less number of pixels than that of the display in order to display the whole image. This is often the case for a display terminal with a very small display as is the case of the cell phone, where the thumbnail image with the resolution 1/64 of the original one is wholly displayed on the display apparatus having a display area of QVGA (320×240 pixels=approximately 80000 (76800) pixels).
That is to say, in such case, the problem is that the resolution of the whole image is decreased, by which the degradation of the image quality is caused.
Another problem is that the part of the image which the user wants to close up (Region Of Interest, e.g., someone's face) becomes relatively small and therefore the quality as well as the is level of visibility is degraded.
In order to solve such problems, it is conceivable to store the original image and display it on the display, however, the problem is that this means to see a part of a large image with a small display apparatus (screen) and imposes strains on the user in the operation. More precisely, the image with XGA size is ten times as big as the display area with QVGA size so that it causes a lot of trouble for the user to scroll in order to find the ROI. In the case where someone's face is shot in a small size, the user needs to zoom-in the part in order to display it. This causes troublesome operations at the side of the user.
Similarly, in the case of listing about nine thumbnail images on the display, the thumbnail images need to be further minimized to the size of 80×60 pixels. In such case, it is again the same problem that the degradation of the level of visibility due to the decrease in resolution for each thumbnail image is evident, and this makes it difficult to see what is in the image.
According to the conventional short film generation apparatus as described above, the visual effect on the image selected beforehand by the user is decided randomly or fixedly for the generation of the scenario necessary for replay. It is therefore impossible to generate a scenario with an impact. That is to say, when deciding a visual effect for each image, the visual effect is assigned in an arbitrary manner without considering what is in the image. Therefore, the trouble is that it is impossible to generate a scenario with an impact, where the characteristics of the image are concerned.
Another problem in using the conventional short film generation apparatus is that it troubles the user because he/she needs to input a title of the image and select the image for generating a scenario.